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SPECIFICATION 

IMAGE COMBINING SYSTEM, IMAGE COMBINING METHOD, AND 
PROGRAM 

5 

TECHNICAL FIELD 
[0001] 

The present invention relates to an image combining system, an 
image combining method, and an image combination program for dividing a 
10 subject into a plurality of partial images, photographing the divided partial 
images, and combining these partial images to generate an image having a 
wider viewing angle and a higher definition. 



BACKGROUND ART 
15 [0002] 

Conventionally, image combining approaches have been known 
for dividing a subject into a plurality of partial images which are 
photographed, and combining the partial images with one another with 
estimation of a positional relationship among the respective partial images to 
2 0 generate an image called a mosaic image which has a wider viewing angle 
and a higher definition. 
[0003] 

For example, Non-Patent Document 1 (Naoki Chiba, Hiroshi 
Kayano, Mitihiko Mino, Shouji Yasuda, "Image Mosaicking Based on Image 
25 Features," Transaction of The Institute of Electronics, Information and 



10. 
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Communication Engineers D-ll, Vol. J82-D-II, No. 10, pp.1 581 -1589) 
discloses a method of generating a mosaic image by estimating a positional 
relationship among respective partial images based on motion vectors within 
the partial images generated by an image analysis. Specifically, sites 
(corresponding points) which are similar in distribution of luminance value 
are found from two partial images which were photographed at nearest times, 
and image conversion parameters are calculated for making a geometrical 
conversion (movement, rotation, scaling up/down) such that all of them 
overlap each other. Then, one partial image is used as a basis to 
geometrically convert the other partial image using the calculated image 
conversion parameters to generate a mosaic image. 
[0004] 

The following model, which supposes that a subject is a flat 
surface, is often used for the image conversion parameters because of its 
simplicity. 
[0005] 
[Equation 1] 
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where (u1 , v1 , 1), (u2, v2, 1) are homogeneous coordinates of respective 
corresponding points, a, b, c, d, e, f, g, h are image conversion parameters, 
and t is an arbitrary constant. 
[0006] 
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Also, for further simplifying the model, a restraint condition may be 
applied to the image conversion parameters, such as g=h=0, a=e=cos 6 , - 
b=d=sin 6 , and the like. 
[0007] 

5 In the approach described in the aforementioned Non-Patent 

Document 1 , a camera must be moved such that an overlapping portion 
exists between two partial images. Otherwise, no corresponding points 
would exist between the partial images, resulting in a failure in calculating the 
image conversion parameters. Also, if part of the subject is not 
10 photographed when the partial images are photographed, a mosaic image 
lacks the partial images, so that a user must exhaustively photograph the 
subject. 
[0008] 

Patent Document 1 (Japanese Patent No. 2744265 (pages 5 - 7, 
15 and Fig. 3) describes an example of an image combining system for 

displaying a screen to guide a user having a camera to take a photograph 
such that the foregoing condition will be satisfied. The configuration of this 
conventional image combining system described in Patent Document 1 is 
illustrated in Fig. 1 . 
20 [0009] 

In Fig, 1, 101 designates a camera body; 102 a shutter button; 
103 a display device, 104 and 105 a switch for instructing whether the 
camera is panned in the left-hand direction or right-hand direction; and 106 a 
lens for focusing a subject on a film or an imager device. 
25 [0010] 
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Fig. 2 illustrates an exemplary display by display device 103 of the 
image combining system illustrated in Fig. 1 . 
[0011] 

For example, when the camera is panned in the right-hand 
5 direction, as the right-hand direction is entered through switch 105, right side 
area 202 of partial image 201 , which has been already photographed, is 
displayed in a left end area of displayed image 203. Further, left side area 
205 of partial image 204 currently captured by the camera is combined in the 
remaining right side screen of displayed image 203 as a moving image, and 
10 the resulting image is displayed. 
[0012] 

The user moves (pans) camera body 101 in parallel with an 
imaging plane, with reference to displayed image 203 on display device 103, 
such that right side area 202 of previously photographed partial image 201 is 

1 5 not discontinuous to left side area 205 of partial image 204 currently 
captured by the camera at the boundary therebetween, and depresses 
shutter button 102 to photograph the partial image. In this way, the mosaic 
image is generated by moving the camera toward a site of the subject not yet 
photographed, while including an overlapping area between the previously 

20 photographed partial image and the partial image currently captured by the 
camera, and sequentially photographing partial images. 
[0013] 

However, in the conventional image combining system described 
in the above-mentioned Patent Document 1 , the camera body must be 
25 moved in the vertical direction and horizontal direction, for example, if an 
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image combining system is manufactured for photographing a subject which 
is broad in both the vertical and horizontal directions, based on the technique 
of Patent Document 1 . Since a direction in which the camera is moved must 
be previously specified by the switch, a problem arises in that the operation 
5 is very complicated. Accordingly, the image combining system described in 
Patent Document 1 is of a configuration which is not suitable for 
photographing a subject which is broad in the vertical and horizontal 
directions. 
[0014] 

10 Also, in the image combining system described in Patent 

Document 1 , the camera body must be moved in parallel with the imaging 
plane, and the amount of movement must be adjusted such that a previously 
photographed partial image is not discontinuous to a partial image currently 
captured by the camera at the boundary therebetween. Thus, the user is 

15 required to perform subtle operations, causing a problem that the user 
experience considerable difficulty taking photographs. 

DISCLOSURE OF THE INVENTION 
[0015] 

20 It is therefore an object of the present invention to provide an 

image combining system and method which are capable of generating a 
mosaic image through simple operations without the need for complicated 
and subtle operations such as manipulations of a switch, parallel movements 
of a camera, alignment between partial images, and the like by the user. 

25 [0016] 
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To achieve the above object, in the present invention, a data 
processing unit calculates image conversion parameters corresponding to 
each partial image, geometrically converts photographed partial images on 
the basis of a current partial image based on the calculated image 
5 conversion parameters, and combines them to generate a mosaic image. 
Also, a display device displays each current partial image and at least part of 
the generated mosaic image. 
[0017] 

In such a configuration, the user is not required to previously 
10 specify, with a switch, a direction in which a camera is moved, or to move the 
camera in parallel with an imaging plane, or to adjust a camera panning to 
avoid a discontinuous image at the boundary of a current partial image with a 
photographed partial image, as required in a conventional image combining 
system. Consequently, a mosaic image can be generated even for a subject 
15 which is broad in the vertical and horizontal directions, through simple 

manipulation, without the need for complicated manipulations or alignment 

by the user. 

[0018] 

Also, in the present invention, each image conversion parameter 
20 is optimized so as to maintain the consistency of the geometric conversion 
and combination among all partial images. Accordingly, shifts in the 
combination are eliminated between partial images which were 
photographed at separate times, thus achieving a mosaic image with less 
distortion. Further, the mosaic image can be generated at a higher 
25 resolution by applying super resolution processing to each partial image. 
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[0019] 

The present invention employs, as a displayed image, an image 
which has a current partial image placed over the entire screen of the display 
device, and a mosaic image generated from photographed partial images 
5 including the current partial image displayed in a portion of the screen at a 
reduced scale; or an image which has a current partial image placed at the 
center of the screen of the display device, and photographed partial images 
included in a predetermined range centered around the current partial image, 
which are combined for display; or an image which has a current partial 
10 image placed over the entire screen of the display device, and a portion 
overlapping with photographed partial images displayed and highlighted. 
[0020] 

As a result, since not only the current partial image but also at 
least part of the mosaic image are displayed on the display device, the user 
15 can readily confirm an area of the subject which has not been photographed, 
and can readily determine in which direction the camera should be next 
moved. Therefore, even when photographing a subject which is broad in the 
vertical and horizontal directions, the user will not forget to photograph part 
of the subject. 

20 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0021] 

[Fig. 1] 

Fig. 1 is a side view illustrating the configuration of a conventional 
25 image combining system. 
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[Fig. 2] 

Fig. 2 is a schematic diagram illustrating an exemplary display by 
the image combining system illustrated in Fig. 1 . 
[Fig. 3] 

5 Fig. 3 is a block diagram illustrating the configuration of a first 

embodiment of an image combining system of the present invention. 
[Fig. 4] 

Fig. 4 is a flow chart illustrating a processing procedure for the 
image combining system illustrated in Fig. 3. 
10 [Fig. 5] 

Fig. 5 is a schematic diagram illustrating a first example of a 
displayed image by the image combining system of the present invention. 
[Fig. 6] 

Fig. 6 is a schematic diagram illustrating a second example of 
15 displayed images by the image combining system of the present invention. 
[Fig. 7] 

Fig. 7 is a schematic diagram illustrating a third example of 
displayed images by the image combining system of the present invention. 
[Fig. 8] 

20 Fig. 8 is a block diagram illustrating the configuration of a second 

embodiment of the image combining system of the present invention. 
[Fig. 9] 

Fig. 9 is a flow chart illustrating a processing procedure for the 
image combining system illustrated in Fig. 8. 
25 [Fig. 10] 
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Fig. 10 is a block diagram illustrating the configuration of a third 
embodiment of the image combining system of the present invention. 
[Fig. 11] 

Fig. 1 1 a block diagram illustrating the configuration of a fourth 
5 embodiment of the image combining system of the present invention. 
[Fig. 12] 

Fig. 12 is a block diagram illustrating the configuration of an 
implementation of the image combining system of the present invention. 

10 BEST MODE FOR CARRYING OUT THE INVENTION 
[0022] 

Next, the present invention will be described with reference to the 

drawings. 
[0023] 

15 (First Embodiment) 

As illustrated in Fig. 3, an image combining system of a first 
embodiment comprises imager device 310 such as a still camera, a video 
camera or the like; data processing unit 300 for combining partial images to 
generate a mosaic image; and display device 320 such as a liquid crystal 

20 panel, a Braun tube or the like for displaying the result of processing by data 
processing unit 300. Data processing unit 300 is implemented by an LSI 
(Large scale Integrated Circuit), a logic circuit or the like which comprises a 
calculation function and a memory. 
[0024] 

25 Data processing unit 300 in the first embodiment comprises 
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conversion parameter estimation unit 301 , storage unit 302, image 
combination data retrieval unit 303, image combination unit 304, and control 
unit 305. 
[0025] 

5 Conversion parameter estimation unit 301 estimates the 

geometric conversion required for a combination of a previously 
photographed (preceding) partial image, among partial images photographed 
by imager device 310, and a partial image which is currently being 
photographed, and calculates image conversion parameters therefor. All the 
10 photographed partial images and the image conversion parameters 

calculated by conversion parameter estimation unit 301 are respectively 

stored in storage unit 302. 

[0026] 

Image combination data retrieval unit 303 selects partial images 
15 required for the generation of a display image in accordance with the 

resolution of display device 320 or a display method (screen layout) specified 
by the user, and retrieves the selected partial images and image conversion 
parameters corresponding thereto from storage unit 302. 
[0027] 

20 Image combination unit 304 geometrically converts the 

photographed partial image on the basis of a current partial image using the 
partial images and image parameters retrieved by image combination data 
retrieval unit 303, and combines them to generate a mosaic image. Also, 
image combination unit 304 generates image data for displaying an image, 

25 which includes the current partial image and at least part of the combined 
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mosaic image, on display device 320 in accordance with the display method 

(screen layout) specified by the user. 

[0028] 

Control unit 305 controls processing by respective conversion 
5 parameter estimation unit 301 , storage unit 302, image combination data 
retrieval unit 303, and image combination unit 304. 
[0029] 

Next, a processing procedure for the image combining system of 
the first embodiment will be described with reference to Fig. 3 using a flow 
10 chart of Fig. 4. 
[0030] 

Assume in the following that the user moves imager device 310, a 
subject is photographed each time the shutter button is depressed or at 
previously set constant time intervals, and photographed partial images are 
15 sequentially supplied from imager device 310 to data processing unit 300. 
[0031] 

As illustrated in Fig. 4, as one partial image of the subject 
photographed by imager device 310 is captured into data processing unit 
300 (step S401), control unit 305 first accesses storage unit 302 to search 

20 whether or not there is a partial image which includes the same subject as 
that photographed immediately before (step S402). If there is a 
photographed partial image, control unit 305 estimates the geometrical 
conversion required for a combination of the current partial image with the 
partial image photographed immediately before that, using conversion 

25 parameter estimation unit 301 , and calculates image conversion parameters 
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required for the geometric conversion (step S403). Then, control unit 305 
stores the current partial image and the calculated image conversion 
parameters in association with each other in storage unit 302 (step S404). 
As a method of calculating the image conversion parameters, for example, 
the method disclosed in the aforementioned Non-Patent Document 1 may be 
used. 
[0032] 

Alternatively, external parameters of the camera may be used for 
the image conversion parameters. In this event, first, in an initial frame, 
feature points are detected on an image, and three-dimensional coordinates 
are determined for each feature point on the assumption that the camera is 
substantially opposite the subject, and the subject is a flat plane positioned 
at a certain distance from the camera. 
[0033] 

In subsequent frames, external parameters of the camera are 
calculated using, for example, an approach disclosed in Tomokazu Sato, 
Masayuki Kanbara, Naokazu Yokoya, and Haruo Takemura, "Three- 
Dimensional Modeling of Outdoor Environment from Moving Image Utilizing 
Multi-Baseline Stereo Method," Technical Report, PRMU2001 -233, February 
2002. This approach estimates the external parameters of the camera by 
tracking predetermined feature points between a currently captured partial 
image and a partial image captured before that. 
[0034] 

If there is no photographed partial image in the search processing 
at step S402, the flow proceeds to processing at step S408, returning to the 
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processing at step S401 , where the next photographed image is captured. 
[0035] 

Subsequently, control unit 305 selects a plurality of partial images 
required for combination into a mosaic image using image combination data 
5 retrieval unit 303, retrieves the selected partial images and conversion 
parameters corresponding thereto from storage unit 302, respectively (step 
S405), and transfers them to image combination unit 304. 
[0036] 

Image combination unit 304 geometrically converts the 
10 photographed partial images on the basis of the current partial image using 
the partial images and image conversion parameters retrieved by image 
combination data retrieval unit 303, and combines them to generate a 
mosaic image (step S406). Image combination unit 304 also sends image 
data for displaying the image, including the current partial image and at least 
15 part of the generated mosaic image, to display device 320 (step S407). 
When the image data is generated for displaying the image, an average 
value or a median value is respectively calculated for each pixel positioned at 
the same coordinates, and these values may be used for pixel values of 
display device 320. 
20 [0037] 

Contemplated as an image displayed on display device 320 is for 
example, a layout which has current partial image 901 placed over the 
entirety of display screen 900, and mosaic image 902, which is a 
combination of photographed partial images, superimposed on part of 
25 display screen 900, for example, in a lower right area at a reduced scale, as 
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illustrated in Fig. 5. 
[0038] 

In this event, for indicating to the user where current partial image 
901 is located, the circumscribed rectangle of current partial image 903 may 
be highlighted within the mosaic image. Non-photographed area 904 which 
does not have a partial image within mosaic image 902 may be identified, for 
example, by setting a particular pixel value (black or the like). Also, when 
current partial image 903 is located outside of mosaic image 902, the entire 
mosaic image may be scrolled and displayed such that partial image 903 fits 
into mosaic image 902, or mosaic image 902 may be displayed at a further 
reduced scale. 
[0039] 

Also, contemplated as a second example of an image displayed 
on display device 320 is a layout which has the current partial image placed 
at the center of the screen at all times, and photographed partial images 
within a predetermined range centered therearound which are displayed in 
combination, as illustrated in Fig. 6. 
[0040] 

In this event, data processing unit 300 first places first partial 
image 1001 entered at time n at the center of screen 1000 at that time (Fig. 
6(a)). 
[0041] 

Next, data processing unit 300 places second partial image 101 1 
entered at time n+1 at the center of screen 1010 at that time, geometrically 
converts first partial image 1001 entered at time n on the basis of second 
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partial image 101 1 , and arranges converted partial image 1012 in 

combination (Fig. 6(b)). 

[0042] 

Similarly, data processing unit 300 places third partial image 1021 
5 entered at time n+2 at the center of screen 1 020 at that time, geometrically 
converts second partial image 101 1 entered at time n+1 on the basis of third 
partial image 1021 , and arranges converted partial image 1022 in 
combination. Further, data processing unit 300 geometrically converts first 
partial image 1001 (or partial image 1012) entered at time n on the basis of 
10 third partial image 1021 , and arranges converted partial image 1023 in 
combination (Fig. 6(c)). 
[0043] 

On the screen, the circumscribed rectangle of partial image 1 001 , 
101 1 , 1021 at each time may be highlighted to indicate the current partial 
15 image to the user. Also, non-photographed area 1002, which does not have 
a partial image present in each screen, may be identified, for example, by 
setting it to a particular pixel value (black or the like). 
[0044] 

Also, contemplated as a third example of an image displayed on 
20 display device 320 is a layout which has a current partial image placed over 
the entire screen, and a highlighted area which overlaps with photographed 
partial images, as illustrated in Fig. 7. 
[0045] 

In this event, data processing unit 300 first places first partial 
25 image 1 101 entered at time n over entire screen 1 100 at that time (Fig. 7(a)). 
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[0046] 

Next, data processing unit 300 places second partial image 1111 
entered at time n+1 over entire screen 1 1 1 0 at that time. Further, data 
processing unit 300 detects overlapping area 1 1 12 of first partial image 1 101 
5 entered at time n and second partial image 1111 from image conversion 
parameters therebetween, and generates an image having highlighted 
overlapping area 1112 (Fig. 7(b)). A method of highlighting overlapping area 
1112 includes, for example, a method of changing the brightness of 
overlapping area 1 1 1 2, a method of blending a particular color in overlapping 
l o area 1112, and the like. 
[0047] 

Likewise, data processing unit 300 places third partial image 1 121 
entered at time n+2 over entire surface 1 1 20 at that time. Further, data 
processing unit 300 detects overlapping area 1 1 22 of second partial image 

15 1111 entered at time n+1 and third partial image 1 121 from image 

conversion parameters therebetween, and generates an image having 
highlighted overlap area 1 122. Further, data processing unit 300 calculates 
an overlapping area of first partial image 1 101 entered at time n with third 
partial image 1 121 from image conversion parameters therebetween, and 

20 generates an image with the highlighted overlapping area (Fig. 7(c)). Here, 
an area in which a plurality of partial images overlap, such as overlapping 
area 1 123, may be highlighted, for example, by performing such processing 
as blending with a deeper color in accordance with the number of partial 
images therein. 

25 [0048] 
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In the third exemplary displayed image illustrated in Fig. 7, a 
mosaic image having not only a current partial image but also partial images 
falling within a predetermined range centered therearound and combined 
thereinto, may be displayed in a portion of the screen at a reduced scale, as 
5 the displayed image illustrated in Fig. 5. 
[0049] 

Which of a variety of the exemplary display images illustrated in 
Figs. 5 - 7 should be displayed on display device 320 may be previously set 
by the user using a switch or the like, or may be switchable in the middle of 
10 photographing. 
[0050] 

As image data is sent from image combination unit 304 to display 
device 320, control unit 305 determines whether or not the processing has 
been completed for all partial images required for combination into the 

15 mosaic image (step S408), and returns to processing at step S401 if 

processing has not been completed to capture the next partial image, and 
repeats the aforementioned processing from step S401 to step S408. 
Conversely, if processing has been completed for all required partial images, 
the mosaic image combination processing is terminated. 

20 [0051] 

Next, description will be given of the effects of the first 
embodiment of the image combining system of the present invention. 
[0052] 

In the image combining system of the first embodiment, the user 
25 does not make alignment between partial images, but data processing unit 



17 



300 geometrically converts partial images for combination based on image 
conversion parameters calculated by conversion parameter estimation unit 
301 , thus making it possible to generate a mosaic image through simple 
manipulations, without the need for complicated switch manipulations, 
parallel movements of the camera, alignment between partial images and 
the like by the user. 
[0053] 

Also, by displaying images as illustrated in Figs. 5-7, not only a 
current partial image, but also photographed partial images are also 
displayed in combination, so that the user can readily confirm a non- 
photographed area of a subject and can readily determine in which direction 
the camera should be moved. In this way, even when photographing a 
subject which is broad in the vertical and horizontal directions, the user will 
not forget to photograph part of the subject. 
[0054] 

Further, since required partial images and image conversion 
parameters are retrieved from storage unit 302 in accordance with the 
resolution of display device 320 or a display image specified by the user to 
generate image data for the display image, the processing for displaying an 
image can be restrained to a minimally required amount. 
[0055] 

(Second Embodiment) 

Next, a second embodiment of the image combining system of the 
present invention will be described using the drawings. 
[0056] 
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As illustrated in Fig. 8, the image combining system of the second 
embodiment, like the first embodiment, comprises imager device 510 such 
as a still camera, a video camera or the like; data processing unit 500 for 
combining partial images to generate a mosaic image; and display device 
520 such as a liquid crystal panel, a Braun tube or the like for displaying the 
result of processing by data processing unit 500. Data processing unit 500 is 
implemented, for example, by an LSI, a logic circuit or the like which 
comprises a calculation function and a memory. 
[0057] 

Data processing unit 500 in the second embodiment comprises 
mosaic image generation unit 501 , and overall optimization unit 502. 
[0058] 

Mosaic image generation unit 501 comprises functions similar to 
those of data processing unit 300 in the first embodiment, and geometrically 
converts and combines photographed partial images on the basis of a 
current partial image to generate a mosaic image. Overall optimization unit 
502 individually optimizes respective image conversion parameters so as to 
maintain the consistency for the results of the geometrical conversion and 
combination among all partial images. 
[0059] 

As a method of optimizing image conversion parameters, the 
following approach is used by way of example. 
[0060] 

First, for feature points in each partial image, all other partial 
images are searched for corresponding points to collect pairs of feature 
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points. Here, an i-th pair of feature points is designated {x\,xi}. 
[0061] 

Subsequently, image conversion parameters between partial 
images are found to minimize the evaluation equation shown below, and 
defined to be final image parameters. 
[0062] 
[Equation 2] 

i 

where 
[0063] 
[Equation 3] 

shows the Euclid distance: 

[0064] 

[Equation 4] 

between point x and point x or between point £' and point x' 

Also, a point 
[0065] 
[Equation 5] 

X 
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indicates the position of point x, the coordinates of which are converted by 

the image conversion parameters. 

[0066] 

When the image conversion parameters are used as external 
parameters for the camera, the image conversion parameters may be found 
such that they minimize the evaluation equation shown below, and they may 
be chosen to be final image conversion parameters. 
[0067] 
[Equation 6] 

YI* w p\ x fp-*fp\ 
f p 

where f is the number of entered partial images, p is the number of tracked 
feature points, xf p is the coordinates of feature point p in an f-th frame, and 

[0068] 
[Equation 7] 

*fP 

is the coordinate on the image of the f-th frame onto which feature point p is 
projected. Also, Wp is a confidence level of the feature point based on a 

tracking error of the feature point. 
[0069] 

For reference, a known Levenberg-Marquardt algorithm (see 
"Numerical Recipes in C [Japanese version] Numerical Calculation Recipes 
in C language," pp.503-507, Gijutsu-Hyoron Co., Ltd, ISBN4-87408-506-1 , 
and the like) may be used for the optimization of the image conversion 



parameters. In this event, values generated by mosaic image generation 
unit 501 may be used for initial values for the image conversion parameters 
corresponding to each partial image. 
[0070] 

Next, a processing procedure for an image combining system of 
the second embodiment will be described with reference to Fig. 8 using a 
flow chart of Fig. 9. 
[0071] 

Assume in the following that the user moves imager device 510, 
and a subject is photographed each time a shutter button is depressed or at 
previously set constant time intervals, and a photographed partial image is 
supplied from imager device 510 to data processing unit 500. Also, assume 
that the following processing by mosaic image generation unit 501 and 
overall optimization unit 502 is controlled by a control unit, not shown, 
possessed by mosaic image generation unit 501 . 
[0072] 

As illustrated in Fig. 9, as one partial image of the subject is 
captured from imager device 510 to data processing unit 500 (step S601), 
mosaic image generation unit 501 generates a mosaic image in a manner 
similar to the first embodiment, and delivers it to display device 520 (step 
S602). 
[0073] 

Upon delivery of image data to display device 502, mosaic image 
generation unit 501 determines whether or not processing has been 
completed for all partial images required for combination into a mosaic image 
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(step S603), and returns to processing at step S601 , if processing has not 
been completed, to repeat processing from step S601 to step S603. 
[0074] 

If processing has been fully completed for required partial images, 
5 overall optimization unit 502 is used to optimize each image conversion 
parameter to eliminate shifts in the combination between partial images 
which were not photographed at adjacent times (step S604). 
[0075] 

Finally, a mosaic image is again generated by geometrically 
10 converting and combining the respective partial images using the image 
conversion parameters optimized in processing at step S604, and the 
mosaic image data is sent to display device 502 (step S605). 
[0076] 

In the foregoing description, the overall optimization processing is 
15 executed for the image conversion parameters after mosaic image 

generation unit 501 has terminated geometrical conversion and combination 
of all partial images, however, overall optimization processing for the image 
conversion parameters may be executed each time geometrical conversion 
and combination processing have been done for a partial image, or may be 
20 executed each time the geometrical conversion and combination processing 
have been completed for every predetermined number of partial images. 
Alternatively, overall optimization processing for the image conversion 
parameters may be executed when a predetermined condition is satisfied, 
such as executed immediately before imager device 510 is largely changed 
25 in the panning direction, and the like. 
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[0077] 

Next, a description will be given of effects of the second 
embodiment of the image combining system of the present invention. 
[0078] 

5 In the image combining system of the first embodiment, image 

conversion parameters are respectively calculated for partial images which 
were photographed at adjacent times, and the partial images are 
geometrically converted in sequence based on these values for combination, 
so that errors of the image conversion parameters may be accumulated to 
10 cause distortion in a mosaic image. 
[0079] 

In the image conversion system of the second embodiment, since 
shifts in the combination are eliminated between partial images which were 
photographed at separate times, the resulting mosaic image can be less 
15 distorted than the first embodiment. 
[0080] 

(Third Embodiment) 

Next, a third embodiment of the image combining system of the 
present invention will be described using the drawings. 
20 [0081] 

As illustrated in Fig. 10, the image combining system of the third 
embodiment, like the first embodiment, comprises imager device 710 such 
as a still camera, a video camera or the like; data processing unit 700 for 
combining partial images to generate a mosaic image; and display device 
25 720 such as a liquid crystal panel, a Braun tube or the like for displaying the 



result of processing by data processing unit 700. Data processing unit 700 is 
implemented, for example, by an LSI, a logic circuit or the like which 
comprises a calculation function and a memory. 
[0082] 

5 Data processing unit 700 of the third embodiment comprises 

mosaic image generation unit 701 , and super resolution image generation 
unit 702. 
[0083] 

Mosaic image generation unit 701 , which comprises similar 
10 functions as those of data processing unit 300 in the first embodiment or 
data processing unit 500 in the second embodiment, geometrically converts 
and combines photographed partial images on the basis of a current partial 
image to generate a mosaic image. Also, mosaic image generation unit 701 
individually optimizes respective image conversion parameters so as to 
15 maintain the consistency for the results of the geometrical conversion and 
combination among all partial images. 
[0084] 

Super resolution image generation unit 702 generates a higher 
resolution mosaic image than the image combining system of the first 
20 embodiment or second embodiment by applying super resolution processing 
to photographed partial images. 
[0085] 

As super resolution processing, the following approach may be 
employed, as disclosed, for example, in Irani Pelog, "Improving Resolution 
25 by Image Registration", CVGIP: Graphical Models and Image Processing, 
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Vol. 53, pp.231 -239, 1991 , or the like. 
[0086] 

Assume now that there exist m partial images {gk}=91 , 92. 9m. 
and assume that f designates a high-resolution mosaic image which one 
wishes to generate. Also, assume that a mosaic image and image 
conversion parameters generated by mosaic image generation unit 702 are 
used for an initial image for mosaic image f , and for respective image 
conversion parameters between partial images {gk}. 

[0087] 

Super resolution image generation unit 702 first sets initial image 
f (0) for a mosaic image, and estimates a low-resolution image (gk(°)> 
corresponding to each partial image {gk} based on the result of simulations 
for the process of geometric conversion and low-resolution mosaic image 
generation including blurring data. 
[0088] 

Next, super resolution image generation unit 702 calculates 
differential image (gk-gk(°)} between the estimated image and entered 

image, and adds pixel values of the differential image to corresponding 
locations in initial image f(0) to generate updated high-resolution mosaic 
image f(1). 
[0089] 

This processing is repeated until error function e indicated by the 
following equation becomes sufficiently small. 
[0090] 
[Equation 8] 
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V * £7) 

where n is the number of times the calculation is repeated. 
[0091] 

5 A mosaic image generated by mosaic image generation unit 701 

may be enlarged, for example, by a factor of two in the vertical and 
horizontal directions, respectively, to create an image for use as the initial 
image for the mosaic image, or may be enlarged by a factor equal to or 
larger than two for use as the initial image. 
10 [0092] 

Next, a description will be given of effects of the image combining 
system of the third embodiment of present invention. 
[0093] 

The image combining system of the third embodiment can provide 
15 a higher resolution mosaic image than the image combining systems of the 
first embodiment and second embodiment by comprising super resolution 
image generation unit 702. 
[0094] 

The image combining system of the third embodiment may 
20 perform only mosaic image combination processing which imposes a 

relatively light processing load (a low calculation cost) while the subject is 
being photographed, and executes super resolution image generation 
processing and overall optimization processing for image conversion 
parameters which impose a relatively heavy processing load (a high 
25 calculation cost) after all partial images have been photographed. In this 



event, a mosaic image update response is prevented from becoming 
degraded while the subject is being photographed. 

[0095] 

5 (Fourth Embodiment) 

Next, a fourth embodiment of the image combining system of the 
present invention will be described using the drawings. 
[0096] 

As illustrated in Fig. 1 1 , the image combining system of the fourth 
10 embodiment, like the first embodiment, comprises imager device 810 such 
as a still camera, a video camera or the like; data processing unit 800 for 
combining partial images to generate a mosaic image; display device 820 
such as a liquid crystal panel, a Braun tube or the like for displaying the 
result of processing by data processing unit 800; and recording medium 830 
15 which records an image combination program for causing data processing 
unit 800 to generate a mosaic image. 
[0097] 

The image combination program is a program for data processing 
unit 800 to implement processing for each conversion parameter estimation 

20 unit 301 , image combination data retrieval unit 303, image combination unit 
304, and control unit 305 included in the image combining system of the first 
embodiment. Alternatively, the image combination program is a program for 
data processing unit 800 to implement processing for each mosaic image 
generation unit 501 , and overall optimization unit 502 included in the image 

25 combining system of the second embodiment. Alternatively, the image 
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combination program is a program for data processing unit 800 to implement 
the processing of mosaic image generation unit 701 , and super resolution 
image generation unit 702 included in the image combining system of the 
third embodiment. 
5 [0098] 

Data processing unit 800 of the fourth embodiment is 
implemented by a computer which has, for example, a CPU, not shown; a 
main storage device for temporarily holding data required for processing of 
the CPU; an I/O unit which is an interface unit with imager device 810, 
10 display device 820, recording medium 830 and the like; and a 

communication device which enables data transmission/reception with a 

server device or the like through a network. 

[0099] 

Recording medium 830 may be a magnetic disk, a semiconductor 
15 memory, a magnetic tape, a CD (compact disk)-ROM, a DVD (digital 
versatile disk), or other recording media. 
[0100] 

Data processing unit 800 loads the main storage device with the 
image combination program recorded on recording medium 830, and 

20 executes processing similar to the data processing units described in the 

aforementioned first to third embodiments by the CPU in accordance with the 
image combination program. The image combination program need not be 
necessarily stored in recording medium 830, but may be stored, for example, 
in a server device or the like on a network and downloaded to the main 

25 storage unit from the server device through a network in response to a 
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request from data processing unit 800. 
[0101] 

Likewise, the image combining system of the fourth embodiment 
can produce effects similar to those of the image combining systems shown 
in the aforementioned first to third embodiments. 

IMPLEMENTATION 
[0102] 

Next, an implementation of the image combining system of the 
present invention will be described using the drawings. 
[0103] 

This implementation is an example in which the image combining 
system of the third embodiment is applied to a mobile telephone system. 
[0104] 

As illustrated in Fig. 12, the mobile telephone system to which the 
image combining system of the present invention is applied, comprises 
mobile telephone 1230, and server device 1240 connected to mobile 
telephone 1 230 through a network. 
[0105] 

Mobile telephone 1230 comprises CCD camera 1210 which is an 
imager device capable of shooting a moving image; processor 1200 which is 
a data processing unit; and liquid crystal panel 1220 which is a display 
device. 
[0106] 

Like the data processing unit of the first embodiment, processor 
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1200 comprises conversion parameter estimation unit 1201, storage unit 
1202, image combination data retrieval unit 1203, image combination unit 
1204, and control unit 1205. 
[0107] 

Server device 1 240 in turn comprises the super resolution overall 
optimization unit shown in the second embodiment, the super resolution 
image generation unit shown in the third embodiment, and a communication 
device, not shown, which enables data communications with mobile 
telephone 1 230 through a network. 
[0108] 

Processor 1200 comprised in mobile telephone 1230 comprises a 
memory, and a CPU for executing processing similar to the data processing 
unit in the first embodiment, for example, in accordance with the image 
combination program. Server device 1240 in turn comprises a memory, and 
a CPU for executing processing for each aforementioned overall optimization 
unit 1241 and super resolution image generation unit 1242 in accordance 
with a program. 
[0109] 

In the mobile telephone system of this implementation, as the 
user moves mobile telephone 1230 while capturing a subject with CCD 
camera 1210, partial images of the subject photographed by CCD camera 
1210 are applied to processor 1200 at previously set constant time intervals. 
[0110] 

As one partial image of the subject is captured, processor 1200 
searches storage unit 1 202 to see whether or not there is a partial image that 



31 



includes the same subject which was photographed immediately before. If 
no partial image is found, the next photographed partial image is captured. 
Alternatively, if there is a photographed partial image, processor 1 200 
estimates the geometrical conversion required for a combination of the 
5 current partial image with the partial image photographed immediately before 
that in accordance with the approach disclosed in the aforementioned Non- 
Patent Document 1 , and calculates image conversion parameters therefor. 
Then, the current partial image and calculated image conversion parameters 
are accumulated in association with each other in storage unit 1202. 
10 [0111] 

Next, processor 1 200 selects a plurality of partial images required 
for combination into a mosaic image, and retrieves the selected partial 
images and image conversion parameters corresponding thereto. In this 
implementation, assume that the second exemplary display image illustrated 
15 in Fig. 6 is displayed on crystal panel 1220. In this event, processor 1200 
retrieves partial images located within a predetermined range centered 
around the current partial image and image conversion parameters 
corresponding thereto from storage unit 1202. 
[0112] 

20 Processor 1200 generates a mosaic image using the retrieved 

partial images and image conversion parameters, and displays the second 
exemplary display image including the mosaic image on liquid crystal panel 
1220. Then, it is determined whether or not processing has been completed 
for all partial images required for combination into the mosaic image, and if 

25 processing has not been completed, processor 1200 retrieves the next 
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image and repeats the processing described above. Conversely, if 
processing has been completed for all required partial images, processor 
1 200 transmits all the partial images and image conversion parameters 
corresponding thereto, as well as a request for processing the mosaic image 
5 (whether or not the overall optimization and super resolution processing are 
required) to server device 1 240 through a network. 
[0113] 

Upon receipt of the partial images and image conversion 
parameters from mobile telephone 1230, server device 1240, in response to 

10 a processing request from the user, optimizes all the image conversion 

parameters so as to maintain consistency after combination into the mosaic 
image by overall optimization unit 1241 in a manner similar to the second 
embodiment, and transmits the optimized image conversion parameters to 
mobile telephone 1 230. In this event, mobile telephone 1 230 again 

15 combines a mosaic image using the image conversion parameters received 
from the server device, and displays it on display device 1220. 
[0114] 

Also, upon receipt of the partial images and image conversion 
parameters from mobile telephone 1230, server device 1240, in response to 

20 the processing request from the user, executes super resolution processing 
for each partial image by super resolution image generation unit 1 242 in a 
manner similar to the third embodiment to generate a high resolution mosaic 
image, and transmits the mosaic image to mobile telephone 1 230. In this 
event, mobile telephone 1 230 displays the high resolution mosaic image 

2 5 received from the server device on display device 1 220. 
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[0115] 

According to the mobile telephone system of this implementation, 
a mosaic image having a wide viewing angle and a high definition can be 
generated using a low resolution camera equipped in mobile telephone 1 230. 
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